Skip to content

Add scalar function performance baselines - #9136

Merged
connortsui20 merged 3 commits into
developfrom
ct/scalar-fn-baselines
Aug 5, 2026
Merged

Add scalar function performance baselines#9136
connortsui20 merged 3 commits into
developfrom
ct/scalar-fn-baselines

Conversation

@connortsui20

@connortsui20 connortsui20 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

Tracking Issue: #9128

Adds stable scalar function performance baselines for byte_length, l2_norm, normalized, inner_product, cosine_similarity, GeoContains, and GeoIntersects.

The benchmark sizes are all comfortably above the roughly 146-instruction Divan harness floor from #9011. Each benchmark binary also uses vendored mimalloc, since scalar function execution allocates its output inside the timed trace and glibc differences between runner images caused the flakes fixed in #8861.

Why

Codspeed can only compare a later implementation change when the same benchmark name already exists on develop, so these baselines need to land before the RowFn things I want to make. And these are pretty cheap, might as well add them.

@codspeed-hq

codspeed-hq Bot commented Aug 1, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 79%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 20 improved benchmarks
✅ 1853 untouched benchmarks
🆕 64 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation div_i64_nonnull 2.4 ms 1.2 ms +96.92%
Simulation add_decimal_i128_nullable 7.4 ms 3.8 ms +96.75%
Simulation mul_i64_nonnull 2.1 ms 1.1 ms +96.57%
Simulation add_i64_nonnull 1,794.2 µs 915.9 µs +95.9%
Simulation add_decimal_i64_nonnull 3 ms 1.5 ms +95.07%
Simulation sub_i64_constant 1,237 µs 639 µs +93.57%
Simulation mul_u32_nonnull 1,391.4 µs 722.4 µs +92.6%
Simulation add_i64_nullable 1,859.1 µs 967.8 µs +92.09%
Simulation lt_i64_nullable 1,242.7 µs 653.4 µs +90.18%
Simulation mul_i32_nonnull 965.6 µs 509.5 µs +89.54%
Simulation mul_u16_nonnull 948.8 µs 502 µs +89.01%
Simulation mul_u8_nonnull 756 µs 405.3 µs +86.53%
Simulation mul_i32_constant 671.6 µs 364.8 µs +84.11%
Simulation mul_i32_nullable 1,032.7 µs 562 µs +83.75%
Simulation eq_i64_constant 680.8 µs 372 µs +83.02%
Simulation mul_i16_nonnull 510.5 µs 281.7 µs +81.19%
Simulation mul_i8_nonnull 289 µs 170.6 µs +69.38%
Simulation and_bool_nullable 186.2 µs 115.4 µs +61.28%
Simulation fsl_large 253.2 µs 211.2 µs +19.92%
Simulation or_bool_constant 78.9 µs 67.4 µs +17.12%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/scalar-fn-baselines (ca01627) with develop (3d0c124)

Open in CodSpeed

@joseph-isaacs

Copy link
Copy Markdown
Contributor

This benchmark has a too long runtime

@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 880ddac to 69edd3a Compare August 3, 2026 15:57
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch 2 times, most recently from f599518 to 9fcb4a7 Compare August 3, 2026 19:56
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch 2 times, most recently from 1dccaaf to bf814bb Compare August 3, 2026 20:30
@connortsui20
connortsui20 changed the base branch from ct/scalar-fn-factory-ext to ct/l2-denorm-encoding August 3, 2026 20:37
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from bf814bb to 003cc85 Compare August 3, 2026 21:19
@connortsui20 connortsui20 added the changelog/performance A performance improvement label Aug 3, 2026
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 003cc85 to 9755708 Compare August 3, 2026 21:40
@connortsui20
connortsui20 marked this pull request as ready for review August 3, 2026 21:58
@joseph-isaacs

Copy link
Copy Markdown
Contributor

Still too long

@connortsui20

Copy link
Copy Markdown
Member Author

@joseph-isaacs what is too long? which one?

https://github.com/vortex-data/vortex/blob/develop/docs/developer-guide/benchmarking.md has nothing about benchmarks being too long, can you make a PR there?

@joseph-isaacs

Copy link
Copy Markdown
Contributor

column_x_column_polygons | N/A | 23.8 ms

@connortsui20

Copy link
Copy Markdown
Member Author

@joseph-isaacs this is unhelpful. There are other benchmarks here that are longer too (i.e. 123ms). Is there some limit here that is undocumented? Why is it bad for microbenchmarks to be "long" in the first place if we're still able to get useful info out of it?

@robert3005

Copy link
Copy Markdown
Contributor

They’re not very micro benchmarks with that runtime. I think in aggregate there’s ~30 non compliant benchmarks. This numbers is an arbitrary threshold we keep to stop them from decaying over time

@joseph-isaacs

Copy link
Copy Markdown
Contributor

I usually think under 1 ms is a good target unless there is a very good reason to go above this.

If we go above this two things happen:

  • the time benchmarks take to run just keep increase at a faster rate
  • codspeed gets upset

connortsui20 pushed a commit that referenced this pull request Aug 4, 2026
…9176)

## Rationale for this change

The benchmarking guide asked for "less than 1ms" per iteration without
saying whether that was an aspiration or a limit, and without saying
which number to measure against it.

That gap showed up in review on #9136, where benchmarks landed at 23.8
ms and 123.4 ms per iteration. The review conversation there had no
documented rule to point at — as noted on that PR, the guide "has
nothing about benchmarks being too long".

This documents the rule first, on its own, so it can be cited
independently of any tooling that enforces it.

## What changes are included in this PR?

Docs only — one section of `docs/developer-guide/benchmarking.md`:

- States that **1 ms is the maximum, not a soft target**.
- Says what to do when a benchmark does not fit: shrink the input, split
it into smaller parameterized cases, or gate it with
`#[cfg(not(codspeed))]`.
- Names the number to check: the **per-iteration** time CodSpeed reports
under `HEAD` for every benchmark a PR adds or changes — not the runtime
of the whole benchmark binary, which is what "this benchmark takes too
long" usually gets confused with.

The heading loses its `~` (`under ~1 ms` → `under 1 ms`), which leaves
the existing anchor `#keep-per-iteration-execution-time-under-1-ms`
unchanged.

A follow-up branch adds CI that reads CodSpeed's report and comments
when a new or changed benchmark exceeds this budget. It is deliberately
kept out of this PR so the rule can be agreed on its own.

## What APIs are changed? Are there any user-facing changes?

No code, no API changes. Documentation only.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn

---
_Generated by [Claude
Code](https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn)_

Co-authored-by: Claude <noreply@anthropic.com>
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 9755708 to 57236cb Compare August 4, 2026 17:28
@connortsui20

Copy link
Copy Markdown
Member Author

@joseph-isaacs everything is sub ms now

@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 57236cb to 210a2c8 Compare August 4, 2026 18:57

@HarukiMoriarty HarukiMoriarty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The geo part micro benchmark looks good to me!

connortsui20 added a commit that referenced this pull request Aug 4, 2026
Measured against develop through #9136's own public-path bench, the RowFn port
was 7.6-7.7x slower across every arm at 65536 rows (13.5us to 104us, divan
fastest and median of 100 samples, best of two ABBA runs, Apple M4 Max). The
non-nullable arms regressed as hard as the nullable ones, so this was never
about masking or validity.

Half of it was a defect worth recording: BytesLen::get called
VarBinViewArray::views once per row, which is a host-buffer downcast, a
vortex_expect whose panic path the optimizer cannot hoist, a length division and
a slice reconstruction. Pointing Varying at the views slice instead took it from
104us to 52us. Bytes::get had the same bug. This is the trap the InputElement::get
doc warns about, and the next person to add an element should assume Varying is
load-bearing rather than decorative.

The other half is not a defect. Develop's views().iter().map(...).collect()
vectorizes to better than one row per cycle, while a strided four-byte field read
widened to eight does not. The row executor itself is fine: row_checked_add hits
0.21ns per row on the same machine, matching both a specialized control and
develop's byte_length, while reading two input columns instead of one. So
byte_length is the outlier, and it is the case #9128 already excludes, where a
row is not the natural unit of work.

Reverting the port leaves Bytes, BytesColumn and BytesLen with no caller, so they
go too, along with the two branch-only byte benches. Shout keeps its coverage as
a test-local element, since it was the crate's only non-dense-safe input and
therefore the only in-crate exercise of the Filter contract and of branch-and-skip
agreeing with filter-and-scatter. The two compile_fail doctests move to inline
elements; both were verified against a positive control, which caught two bugs in
the rewrite where they had been failing for unrelated reasons.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Base automatically changed from ct/l2-denorm-encoding to develop August 5, 2026 15:01
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 210a2c8 to f0d133b Compare August 5, 2026 15:01
Comment thread vortex-array/benches/byte_length.rs Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this add 8 benchmarks, I am not sure we need this.

Also it only benchmarks all inline or all not inline

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im happy to remove this, though we should decide if we want to rewrite a scalar function like this with rowfn or not. It is possible, and it is also possible to get close to parity, but maybe not worth it.

Comment thread vortex-array/benches/byte_length.rs Outdated
Comment on lines +93 to +98
let validity = Validity::from_iter((0..len).map(|i| i.is_multiple_of(10)));
let input = MaskedArray::try_new(long_strings(len), validity)
.unwrap()
.into_array();
bench_byte_length(bencher, input);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

measures masking and str len

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want a like expr and we don't want row fn like lifting

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait why not? this seems like a good place to do it since it is localized compute with one setup (compile dfa/regex)?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cause i want to run the dfa over the whole string batch at once

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it just for simplification, but when you say "whole string batch at once" you still have to do it sequentially. And ideally the rowfn machinery should get optimized away so that it looks like its the same

fn execute(distance: VortexResult<impl IntoArray>, ctx: &mut ExecutionCtx) -> ArrayRef {
distance
.unwrap()
.into_array()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want this in the benchmark

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think we want this

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the into_array. I guess that is just a Arc::new()

Comment thread vortex-geo/benches/envelope.rs Outdated
.bench_values(|(array, mut ctx)| {
array
.into_array()
.execute::<ExtensionArray>(&mut ctx)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this do a full canonnical?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whoops

we really should change this

@connortsui20 connortsui20 Aug 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh wait no it does do stuff because the input is not an extension array, it is a normalized array

but we should change the extension array execute regardless...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes we should

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20

Copy link
Copy Markdown
Member Author

whoops I deleted like.rs, will bring it back without modification

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 merged commit 9a482c0 into develop Aug 5, 2026
92 checks passed
@connortsui20
connortsui20 deleted the ct/scalar-fn-baselines branch August 5, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants